Protocol for precise signal synchronization of electrophysiology, videography, and audio recordings using a custom-made pulse generator

Summary Precise signal synchronization is vital for accurate analysis in systems neuroscience. Here, we present a protocol for synchronizing electrophysiology, videography, and audio recordings using a custom-made pulse generator. We describe steps for building the pulse generator, installing software, connecting devices, and running experimental sessions. We then detail signal analysis, temporal alignment, and duration normalization. This protocol offers flexibility and cost-effectiveness, addressing limited shared knowledge and providing a solution for signal synchronization in various experimental setups.

There are commercially available high quality pulse generators such as Master 8 (AMPI) and Multistim 3800 (A-M systems), whereas low-cost open source pulse generators such as Pulse Pal 1 have become available recently. In our particular case, we had specific requirements in addition to a budget limitation: (a) we needed only monophasic digital input/output; (b) we needed 5 independent channels; (c) output in each channel should be triggered by either pressing a button, external digital input, or MATLAB function. Most commercially available pulse generators do not have trigger input for each channel. Thus, we developed a custom pulse generator using Arduino and Teensy technology ( Figure 1A). The Teensy board serves the main pulse generator function, whereas the Arduino board works as an interface between the Teensy board and MATLAB. This custom-designed solution offers 5 independent channels. Pulses are triggered by either external TTL input, push button, or MATLAB function via Arduino. Each of the five channels offers four different pulse modes: pulse, toggle, asinput and togglePulse ( Table 1). The input is 5 V tolerant TTL, and the output is 3.3 V TTL. To streamline the building process, we designed a printed circuit board (PCB) of the pulse generator ( Figures 1B and 1C). Note: When connecting the LEDs, pay attention to the polarity: connect the pins with the ''+'' signs on the PCB to the longer wires of the LEDs. There is no polarity for switch and button connections. Since necessary resistors are already installed on the PCB, it is unnecessary to connect further resistors for LEDs and switches.
Note: Panel mount power switch with LED indicator may have 3 or 4 connectors. See the datasheet of the purchased switch to confirm the role of each connector.
Note: You may need to solder pin headers to connectors of switches, BNC connectors and push buttons for connections with jumper wires.
Alternatives: Rack-mount housing is not necessary for the function of the pulse generator, but it substantially facilitates practicality of the operation.
4. Connect the two USB cables of the pulse generator housing to a computer ( Figure 1B).

Pulse generator software settings (standalone)
Timing: 20 min You first need to configure a custom sketch on the Teensy board to set pulse settings. This can be achieved by using the Arduino Integrated Development Environment (IDE) and uploading the provided example sketch, ''myPulses.ino'', available on the github repository at https://github.com/ neurogelotology/synchronization_protocol/tree/main/Pulse_Generator_software.

Open
Arduino IDE, Sketch > Include Library > Add .Zip Library, and select ''pulseGen.zip''. 6. Turn on the main switch of the pulse generator. 7. Setup connection of the Teensy board. a. Click on ''Tools'' in the main menu and select ''Port'' from the drop-down list. b. A list of available serial ports will appear. The Teensy should be listed as ''Teensy'' followed by the PORT number (e.g., ''Teensy (COM3)''). c. Select the correct PORT. 14. Test the pulse generator by pressing trigger buttons.
Note: The output pulses are indicated by the output LED on each channel. Pressing the STOP button will stop all running pulses.
Optional: You can monitor the pulse output using an oscilloscope. To do so, connect the oscilloscope probe to the BNC output connector on the pulse generator. You can then observe the pulse frequency and the voltage level on the oscilloscope. This will help you verify that the pulse generator is functioning correctly and outputting the desired pulses.
Pulse generator software settings (MATLAB trigger)

Timing: 3 min
With MATLAB integration, users can design more flexible trigger patterns for the pulse generator. Specifically, MATLAB triggers a pulse output in the Arduino in the pulse generator, which in turn trigger a channel of the main Teensy board. The pulse generator can be controlled using four MATLAB functions (Table 2; available at https://github.com/neurogelotology/synchronization_ protocol/tree/main/Pulse_Generator_software).
15. Turn on both the main and Arduino power switches on the pulse generator ( Figure 1A). 16. Restart MATLAB. 17. Type. in the command window and press Enter. It should display ''Nano 3 is connected'' in the command window.
CRITICAL: If connectArduino does not generate an Arduino instance, please refer to problem 1.

Try executing,
which sends a pulse to the Ch1 input of the pulse generator. If you configured the togglePulse mode in the previous preparation, the Ch1 output LED should start blinking at 1 Hz.
CRITICAL: Since output of the Teensy 3.5 board is 3.3 V, the pulse generator may not be able to trigger devices that requires 5 V TTL input. It is possible to convert 3.3 V TTL to 5 V TTL using a level shifter circuit, which, however, might cause delays in the timing of the pulses.
Optional: In addition to signal synchronization, the pulse generator can be used for a variety of applications including triggering simulations in a stimulus isolator, triggering optogenetic stimulations, switching environmental illumination, and activating audio visual stimuli. Each channel of the pulse generator works independently. For instance, users can design specific behavioral paradigms where optogenetic stimulations are scheduled at certain timings. In such cases, the pulse generator can be used to trigger optogenetic stimulations, and the timing of the stimulations can be controlled in MATLAB. Example MATLAB function to trigger sync pulses and stimulations is available at https://github.com/neurogelotology/ synchronization_protocol/blob/main/Pulse_Generator_software/exampleTimer.m.

Hardware settings
Timing: 1-3 h The goal of this section is to establish connections between all devices. To demonstrate this process, we use LabLynx Portable System (Neuralynx) for electrophysiological recordings, UltraSoundGate 116Un (Avisoft) for audio recordings, and two BFS-U3-28S5C-C cameras (FLIR) for behavioral videography. A comprehensive overview of the device wiring is depicted in Figure 2A.
19. Connect Ch1 output of the pulse generator to the digital I/O connector on LabLynx ( Figure 2B, left) using a BNC-AL 0.64 socket cable. The black GND socket should be connected to the lower pin of the LabLynx. 20. Construct a cable with a BNC male connector on one end, and a 2.5 mm TS connector on the other. a. Cut a BNC male cable and a 2.5 mm TS cable in the middle. b. Strip the insulation of the two cables. Separate the core wire and the shield wires. c. Insert heat shrink tubes on one of the cables and the core wire. d. Solder the core wires of the two cables together, and apply a heat gun to the heat shrink to ensure proper insulation of the core wire. e. Solder the shield wires of the two cables together. f. Apply a heat gun to the heat shrink covering the cable. 21. Test connectivity using a multimeter.
a. Set the multimeter to continuity mode, which is typically with a diode symbol with an audio feedback feature. In continuity mode, the multimeter will beep if there is an electrical connection between the two test probes.
Test the connection between the BNC pin and the tip of the TS jack. The multimeter should beep if the connection is good. c. Next, test the connection between the BNC shield and the sleeve of the TS jack. The multimeter should beep if the connection is good. d. Finally, test the insulation between the BNC tip and the sleeve of the TS jack. There should be no beep, indicating that there is no electrical connection between these two points.
CRITICAL: Whenever constructing a custom cable using soldering, it is always recommended to test the connection.
22. Connect Ch1 output of the pulse generator to the DIN socket on the UltraSoundGate using the BNC-TS cable ( Figure 2B, right). BNC distributors (T-or Y-shaped) can be used to split the pulse signals from the pulse generator. 23. Construct two cables that have a GPIO (Hirose HR10) connector on one end and a mini din6 male connector on the other. Note: Ensure the wires are soldered properly to the corresponding pins of each connector, as depicted in Figure 2C. See steps 20-21 for soldering, insulation and testing the connections.
24. Assemble a camera connection box housing with two panel mount mini din6 female sockets and a BNC female connector, and solder the wires to the appropriate pins according to the wiring diagram ( Figures 2D-2F). 25. Connect the camera connection box and two cameras using the GPIO-Mini Din6 cables, and connect the camera connection box and the pulse generator Ch1 output with a BNC cable ( Figure 2A).
Note: The camera connection box that serves as a hub for managing signals from the two cameras and the pulse generator. In specific, using the connection box, pulses from the pulse generator are sent to both cameras. Moreover, exposures between the two cameras can be synchronized. The diagram in Figure 2D provides a visual representation of the wiring in the connection box. Here we use commercially available mini din6 cables to interface between the camera connection box and the GPIO (General Purpose Input/Output) cables that connect to the cameras ( Figure 2C).
Alternatives: Soldering all connections directly may work, but having a connection box makes the setup more organized and convenient to use in practical settings. The connection box allows for a clear and easy way to connect and disconnect the cameras and the pulse generator. Additionally, having a connection box can make the setup more portable, which can be easily disconnected and transported to a different location.
CRITICAL: Having synchronized capture is crucial for accurate multi-camera video analysis, as it ensures that the exposure timing of both cameras is perfectly aligned.
Note: The cameras used in this setup offer triggered exposure, meaning that an exposure of the primary camera will trigger an exposure of the secondary camera, thereby ensuring that the exposures are synchronized. For technical details about synchronized capture, readers are encouraged to consult the blog post by FLIR.
CRITICAL: GPIO connections ( Figure 2D) for synchronized exposure and logging external TTL input may vary between camera models and camera manufacturers. To ensure proper wiring of the connection box, it is crucial to refer to the specific details provided in the user manual of the camera being used.
Alternatives: Signal synchronization of video using a camera that does not accept external TTL input such as a webcam or a camcorder can be achieved by placing an LED visible to the camera. The anode (long pin) of the LED should be connected to the core pin of the pulse generator's Ch1 output, while the cathode (short pin) should be connected to a 220 Ohm resistor which in turn should be connected to the shield of the pulse generator's Ch1 output. The LED method for camera synchronization has limitations in terms of sensitivity to ambient lighting conditions, and complexity in ensuring a clear field of view, making it less suitable for applications involving challenging lighting conditions or multiple cameras.

Camera software settings (SpinView)
Timing: 20 min In this section, we demonstrate how to configure the settings of the camera software. Specifically, we use the SpinView software for two BFS-U3-28S5C-C cameras by FLIR as an example. sync TTL pulse can be monitored real time so that users can notice when the system is not working properly.
33. Open Bonsai. 34. File > Open, then select TwoCam.bonsai file (Figure 3). 35. Set file names ( Figure 3A). a. Click on the top right ''VideoWriter'' node with a camera image and purple background. b. In the Properties panel at right, click ''FileName'' and click on the ''...'' button on the right. c. Navigate the destination folder and type the primary video file name that ends with ''.avi''. d. Next, click on the purple ''I/O'' node below the primary camera node. e. Similarly, set the file name for the timestamp file that ends with ''.csv''. f. Similarly, set the video file name and the timestamp file name for the secondary camera below. 36. To ensure that both cameras are capturing images correctly, click ''Start'' in the top left ( Figure 3B).
Note: Verify that the frame ID is indicated on each image. Additionally, if the pulse generator is in operation, the TTL state should be displayed in real time. If the interval between frames is longer than double of the expected inter-frame interval, it will be counted as a dropped frame.
CRITICAL: If bonsai crashes upon ''Start'', the Spinnaker SDK version may not be compatible with the Spinnaker Library of bonsai. Try installing different Spinnaker SDK versions.
CRITICAL: If the dropped frame counter keeps increasing, refer to problem 2.

Audio recording software settings (Avisoft RECORDER)
Timing: 10 min Here we focus on the settings regarding the TTL sync pulses in Avisoft RECORDER software. In contrast to video recordings where the timestamp and TTL state of each frame are saved in a separate CSV file, Avisoft RECORDER saves external TTL states in the least significant bit (LSB) of the audio signal within the .wav file. Official manual of the RECORDER can be found here.
Note: Avisoft RECORDER offers various options for handling external TTL inputs, such as triggering recording start with a TTL pulse. In this demonstration, we will manually start and stop the recording without using the triggered record mode. Note: With this configuration, the external TTL pulses from the pulse generator will not trigger anything, but will be saved in the LSB of the .wav file. Furthermore, the external TTL states will be indicated as blue bars on the top of the spectrogram.

Recording experiment
Timing: Depends on experimental paradigm (a few minutes to hours) We will perform recording of electrophysiology, ultrasound and videos, while sending a synchronization TTL to the recording devices ( Figure 4A). 45. Open Cheetah (Neuralynx) software. Click on the ''ACQ'' and the ''REC'' to start recording electrophysiology. 46. Open Avisoft RECORDER (Avisoft) software. Click the window to start recording audio.
Note: If the software is not launched because the driver is not found, see problem 3.
47. Open bonsai software. Set four file names as described in the step 35 of the preparation, and click the ''Start'' button to start recording videos. 48. Press the Ch1 trigger button of the pulse generator to start the sync pulse train. Verify that each device is properly receiving the TTL pulses. a. On Cheetah, new TTL events are listed in the ''Events'' window. b. On Avisoft RECORDER, the TTL status is indicated by blue lines at the top of the spectrogram. c. On bonsai, the TTL status is plotted on a separate window for each camera ( Figure 3B). 49. Start experiment e.g., behavioral paradigm.
CRITICAL: Throughout the recording session, it is important to ensure that the TTL pulses are being delivered to each device and that the drop frame counters in bonsai are not increasing. Monitoring these parameters can help identify issues and prevent data loss. If you encounter a continuous increase in the drop frame counters, refer to problem 2 for a possible solution.
50. After completion of the experiment, press the Ch1 trigger button of the pulse generator to stop the sync pulse train. CRITICAL: Sync pulse train must start only after all the devices start recording, and stop before any device stops recording to ensure that the same number of pulses are recorded in all devices ( Figure 4A).

Signal alignment
Timing: 30 min after signal analysis After a recording session has finished, each signal is analyzed typically to produce time vectors or continuous time series ( Figure 4B). For example, extracellular electrophysiological signals can be analyzed using spike sorting software to produce spike times for multiple single units. Audio analysis can yield time points of vocalizations. Behavioral analysis of video data can produce start and stop times for different behavioral events. Furthermore, times of the sync pulses in each signal will be extracted as a time vector. In this section, we demonstrate how to align time vectors resulted from different devices.

Complete analysis of each signal.
Note: We do not cover each signal analyses in this protocol. The results are expected to be time vectors or continuous time series that may accompany additional data. For example, Figure 4. Workflow of the signal synchronization (A) A recording session yields electrophysiology data (yellow), audio data (light green), and video data (film). While acquiring signals from these devices, sync pulse train (red) is sent to the devices from the pulse generator. Experiment is performed while the sync pulse train is being sent. (B) After the recording session, each signal will be analyzed to acquire time vectors of events. Spike sorting is performed to acquire spike times from the electrophysiology data. Vocalization analysis is performed to acquire vocalization times from the audio data. Start and stop times of behaviors are detected in the video analysis. Moreover, sync pulse times (red ticks) are extracted in each signal. To ensure that all the sync pulses were saved in each signal, compare the number of pulses and inter-pulse intervals between the signals (see Figure 4). 56. Verify the sync pulses in each device. This involves checking whether the numbers of pulses are equal across all devices, and whether the inter-pulse intervals are stable.
Note: Figures 5A-5D and Table 3 show verification of sync pulses in an example recording session. Further details of the sync pulse analysis are described in overview of the sync pulse analysis.
57. Align the time vector data from different signals such as spike times, vocalizations times and event times, using the sync pulses. Alignment of the signals can be achieved by subtracting the time of the first sync pulse from all time vector data for each device, resulting in time vectors relative to the first sync pulse time ( Figure 4C).
Note: Time values in the time vector data can be either as timestamps of the recording device, or times relative to the first sample time of the recording, dependent on the analysis algorithm. In the latter case, difference between the first sample timestamp and the first sync pulse rise timestamp must be subtracted from the time vector data.

EXPECTED OUTCOMES
Using the signal synchronization methods we have introduced in this protocol, signals recorded in different devices are temporally aligned ( Figure 4D). Temporally aligned data can be used to plot peristimulus time histogram to analyze relationships between different signals, such as neuronal firing rate or vocalization rate upon a behavioral onset. [6][7][8][9] QUANTIFICATION AND STATISTICAL ANALYSIS Overview of the sync pulse analysis We evaluated the performance of our custom pulse generator by analyzing the sync pulses recorded by LabLynx, Avisoft and two FLIR cameras during a 10.5-min recording session (Table 3). All 630 sync pulses were detected in all devices, and the session duration (last pulse time -first pulse time) was on average 628.9969 G 0.0044 s (mean G SEM). To account for differences in session durations between devices, we describe a normalization method in the next section. Inter-pulse interval of the sync pulse, which is theoretically 1 s constantly, was stable both in LabLynx ( Figure 5A) and Avisoft ( Figure 5B), where the fluctuation was smaller than G0.1 ms. The video devices periodically showed small inter-pulse intervals ( Figures 5C and 5D) due to clock mismatch between the pulse generator and the cameras, which we explain in ''artifacts due to the clock mismatch'' section. Overall, our protocol using the custom pulse generator offers highly precise signal synchronization between multiple recording devices.
Normalization of the session durations Due to differences in hardware clocks, time data from different recording devices are measured in different timescales. In other words, 1 s in one device may not have the same duration as 1 s in another device (Figures 5A and 5B). This clock difference is typically consistent over time, resulting in a linear drifting effect when the time difference between sync pulse times from different devices is plotted over pulse index ( Figure 5E, ''Raw''). To analyze the relationship of data recorded in different devices, this linear drifting needs to be corrected. To achieve this, session duration (last pulse timefirst pulse time) recorded in each device needs to be normalized to the session duration recorded in a reference device ( Figure 5E, ''Normalized''). In our example, we select the electrophysiology as the reference device because of the high sampling rate (30 kHz), whereas the videos' frame rate is 30 fps. Additionally, the time information of the electrophysiology data is given as the device's clock timestamp. Although the audio data has a higher sampling rate of 256 kHz, the audio data time is given as the time in .wav file with a fixed (ideal) sampling rate and not the device's clock timestamp, which could be fluctuating in reality (Figures 5A and 5B). The normalization can be performed as mentioned in the ''signal alignment'' section. Artifacts due to the clock mismatch In Figures 5C and 5D, we present the inter-pulse intervals of sync pulses recorded in the two cameras. In addition to negligible fluctuations in the sampling rate ( Figure 5D inset), the inter-pulse intervals show a periodical decrease of 0.0332 s, which is precisely the inter-frame interval of the video recordings. Through a simulation, we verified that the periodical decrease in the inter-pulse interval was caused by clock mismatch between the camera and the Teensy board of the pulse generator. In our simulation, we set the camera clock to run 0.01% slower than the clock of the pulse generator, indicating that 1 s in the camera is equivalent to 1.0001 s in the pulse generator. Additionally, we set the camera exposure time to 0.015 s for each frame, with a frame rate of 30 fps. The simulated interpulse interval, shown in Figure 5F, closely matches the actual data observed in Figures 5C and 5D. Consequently, we conclude that the periodical decrease in the inter-pulse interval is not indicative of frame drop, but rather a result of clock mismatch between the cameras and the pulse generator. By normalizing the duration as described above, the video data can be synchronized with signals from other devices. The simulation of the clock mismatch is available at https://github.com/ neurogelotology/synchronization_protocol/blob/main/syncTTL/sync_pulse_analysis.mlx.

LIMITATIONS
While the protocol presented in this article offers an effective solution for precise signal synchronization, there are several limitations to its applicability.
Device specificity: This protocol focuses on specific devices and software settings as examples, and may not be directly applicable to all recording setups. The general concept can be adapted to other similar devices and software, but users must verify compatibility and adjust settings accordingly. It should be noted that some recording devices such as conventional microphones and webcams are not capable of receiving digital inputs.
Data format specificity: The format of the data may vary between recording devices and analysis software. The users may need to adjust the analysis functions we provide.
Pulse generator dependency: This protocol partly relies on the use of our custom-made pulse generator for signal synchronization, which may not be applicable for all users. Alternatively, any common pulse generators can be used.
Recording environment: The accuracy of the signal synchronization depends on the stability of the recording environment including the recording devices and the computers.
Limitation of temporal precision: Our protocol may not provide the level of temporal precision required for research that demands sub-millisecond accuracy. In such cases, more advanced synchronization techniques may be necessary.

TROUBLESHOOTING
In the Troubleshooting section, we provide solutions for common problems that can occur repeatedly during experiments. For problems that may arise during the setup phase, we have outlined potential solutions in the before you begin section and the step-by-step method details section.

Problem 1
connectArduino does not generate an Arduino instance in MATLAB (typically occurs in pulse generator software settings (MATLAB trigger) and recording experiment). You should see a new COM port added to the list. Note the COM pote number. If the Arduino is COM3, for example, execute, in the Command Window of MATLAB, which uploads server code in the Arduino. This issue can happen when you first operate the pulse generator from the MATLAB, or change the USB port on the computer.

Problem 2
The dropped frame counter on bonsai keeps increasing (typically occurs in video recording software settings (bonsai) and recording experiment).

Potential solution
Here are some possible causes to consider: Overfilled memory cache: If the computer's memory cache is full, the recording may not function properly. Stop the recording, and restart the computer. Storage space: Make sure to empty the storage and free up some space at least 15% of the storage capacity as a common practice. Too high frame rate: If the frame rate is too high for the computer's processing power, it can lead to recording problems. Decreasing the frame rate can resolve this issue. Slow writing speed: If the writing speed of the storage device is not fast enough, it can lead to frame dropping. Consider using a high-speed SSD, specifically one that is not the system drive for data storage. Avoid saving on a network drive.

Problem 3
Avisoft RECORDER fails to start because the driver is not found (typically occurs in audio recording software settings (Avisoft RECORDER) and recording experiment).

Potential solution
Reinstall the driver (Windows).
Uninstall the driver.
Open Device Manager. The Avisoft device may be recognized under either ''Sound, video and game controllers'', ''Universal Serial Bus controllers'', or ''Avisoft-UltraSoundGate'' category. If it is under ''Universal Serial Bus controllers'', the device name is ''USB Composite Device''. Right click on the Avisoft device, and click ''Uninstall''. Reinstall the driver.
Once the driver is uninstalled, disconnect and connect the Avisoft device again. Windows will automatically install a driver and assign the device as ''Audio Endpoint''. This driver needs to be uninstalled by right clicking and clicking ''Uninstall''. Then the device will become a ''USB Composite Device''.